Information Theory Tutorial Communication over Channels with memory

A general capacity formula C = sup X I(X; Y), which is correct for arbitrary single-user channels without feedback, is introduced in this tutorial. This new capacity formula is obtained by using a general channel model without any assumptions of the channel, introducing the notion of inf/sup-information/entropy rates, and finding a new tight converse bound on the error probability. In Section 1 we will show the error of the conventional channel capacity formula. Then in section2, we will give some definitions used in the proofs of the capacity formula. Section 3 is devoted to the direct coding theorem. Section 4 will focus on the new converse bound and the general capacity formula. We will see some useful results of the new converse bound in section 5. In section 6, we will see how feedback can increase the channel capacity when the channel has memory.